Demo project integrating ScrapeGraphAI SDK with Elasticsearch to enable intelligent product scraping, indexing, and cross-marketplace comparison.

Core Implementation

• Configuration (src/scrapegraph_demo/config.py) - Environment-based config with Elasticsearch/API key management
• Data Models (src/scrapegraph_demo/models.py) - Pydantic models for Product and ProductComparison with validation and serialization
• Elasticsearch Client (src/scrapegraph_demo/elasticsearch_client.py) - Index management, bulk operations, full-text search with filters, aggregations
• Scraper (src/scrapegraph_demo/scraper.py) - ScrapeGraphAI integration with mock data fallback for testing

Features

• Multi-marketplace product scraping (Amazon, eBay, etc.)
• Full-text search with fuzzy matching, price range, and marketplace filters
• Product comparison: find cheapest, best-rated, group by marketplace
• Price statistics and aggregations

Infrastructure

• Docker Compose setup for Elasticsearch + Kibana
• 12 unit tests with 100% pass rate
• 3 example scripts demonstrating usage patterns
• Interactive quickstart demo

Usage Example

from src.scrapegraph_demo import Config, ElasticsearchClient, MarketplaceScraper

config = Config.from_env()
es_client = ElasticsearchClient(config)
scraper = MarketplaceScraper(config)

# Scrape and index products
products = scraper.scrape_search_results("laptop", "Amazon", max_results=10)
es_client.index_products(products)

# Search with filters
results = es_client.search_products(
    query="laptop",
    min_price=500,
    max_price=1500,
    marketplace="Amazon"
)

# Compare across marketplaces
comparison = ProductComparison(query="laptop", products=all_products)
cheapest = comparison.get_cheapest()
best_rated = comparison.get_best_rated()

Security: No vulnerabilities in dependencies (verified with gh-advisory-database and CodeQL)

• Fixes Implement ScrapeGraphAI SDK Integration with Elasticsearch for Marketplace Demo #1

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.